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ABSTRACT 

The role of the evaluation supervisor as internal 
auditor and his or her responsibility for assuring the independence 
and objectivity of evaluation results are discussed. Four sources of 
compromise of objectivity are discussed: factual, political, 
involvement^ and affective. The first two sources of compromise 
cannot be accepted, A code of ethical evaluation practices is needed 
to deal vith them. The second tvo sources of compromise ari,se from 
people* s relating to other people. Knowledge of these relationships 
enables the evaluation supervisor to deal as an auditor with effects. 
The relationship of the evaluation supervisor to the evaluator and 
the program manager are discussed. (Author) 
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Once the researcher Icoves the area of the physical sciences, he must 
forego the luxury of studying subjects v/hich can be kept at "standard 
tenperature and pressure;" once he begins studying subjects who can "stare 
back" and interact with iiim, he nust exc":nine his own feelings about them and 
question his own objectivi ty.^ Being hu'^an and, hopefully, reasonably normal, 
he, too, has ego needs. These can influence the v/ay he vie\/s things. It is 
a truisn that: A glass of water can be either half full or half empty. The 
perspective from which an evaluator "looks" can, of course, have an impact on 
what he "sees." Should the researcher be in the position of having his 
livelihood depend on how well the subject studied likes what is said about 
bin, then he is in a precarious position, indeed. 

The above is merely to provide some perspective for a consideration of 
the position of evaluation services in a city school system. Essentially, 
one is faced with a paradox: to be useful the evaluator must be familiar with 
the project studied, the setting in which it is operating, and the people 
involved in it; to be objective he must be detached^ The evaluator has to be 
"close enough to knav you and distant enough to see you." 

Responsibility for dealing with this problem (i.e., responsibility for 
assuring the objectivity of the evaluation) can, in my opinion, best be 
assumed by the evaluation supervisor, since he is familiar with the evaluation 
and with the system in which it is being conducted. As used in this paper, 
the term evaluation supervisor refers to the individual in line above the 
evaluator and to whom the evaluator reports. In short, it refers to the 
individual v/ho must "sign off" on a report before it is published. Depending 
on the size of the organization, there can be more than one level of audit. 

Perhaps my reasoning for the above opinion might best be illustrated by 
the follov/ing true incident, with only the names deleted to protect the guilty. 
A number of years ago, several distinguished faculty members from an equally 
distinguished university contracted to evaluate a large innovative program. 
One of their conclusions was that class size varied directly with achievement. 
That is, the larger the class size, the better the pupil performance. They, 
too, v/ere surprised by this finding, and after further investigation discovered 
what thousands of professional employes already knew: classes for slow learners 
were purposely made smaller. One might reasonably wonder what other character- 
istics of the system were ignored in the evaluation. 

To address the problen of assuring objectivity in evaluation, one must 
first identify the possible sources of compromise of objectivity. Techniques 
can then be developed to deal v/ith each. 

It is suggested that there are four basic sources of compromise of objec- 
tivity. Each will be discussed and ways of responding to it will be proposed. 

1 . Factual Conp ro:r> i s e 

This arises v/hen the needs of the project director and the interests 
of t^^e evaluator cannot be separated. It arises when the evaluator is on the 
budget of the project and reports directly to the project director. The 
continuation of the project - and therefore of the evaluator^s job - may depend 
on a favorable report of project achievement. 



In sucii a case, the evaluator has in fact been compronised. There is 
no way in v/hich he can prevent his personnl interests and h\s professional 
integrity from coming into conflict unless - happily - he is evaluating the: 
gold producing powers of the Philosopher's Stone. 

There is really no solution to this situation except to keep away from 
it. The Research Office should not become involved with any evaluation 
unless it is clearly understood that all evaluation personnel report directly 
to the Office of Research and Evaluation and that the office rather than the 
individual is responsible for the evaluation. 

Various mechani sr.s for effecting this are in use. In some cases, evalua- 
tion funds are transferred to the budget of the Research Office. In other 
cases they are merely ear^^arked for evaluation and can be spent in no other v;ay. 
The fnain point is that either way the evaluator must see himself as working 
for the Research Office (which usually has other programs to evaluate should 
the current one be discontinued) and not for the project director. The project 
is the client, and not the boss I 

A yet more viable approach was recently taken by the Department of Public 
Welfare of the State of Pennsylvania. After receiving what they considered 
satisfactory assurances of the independence of the Research Office from the 
various Instructional Divisions of The School District of Philadelphia, they 
insisted on contracting directly with the Research Office for the evaluation 
of the Day Care Services Program for which the School District is prime 
contractor. This move accomplished three ends: it utilized evaluators and 
evaluation supervisors who either were or had the opportunity to become 
completely familiar with the system; it made the Evaluation Unit literally 
independent of the Program Unit for its funding; and it placed the State in 
the position of beinc able to hold the Research Office direc'tly accountable 
for the contracted evaluation services (i.e., the State became the client). 

Aside from the above, the only possible way around the problem is to 
insist on the use of an outside evaluator retained on a consultant basis, in 
the hope that the consultant fee will be a small enough part of his total 
incone that hev/ill be unwilling to compromise his professional integrity for 
it. 

2. Pol i tica 1 Corr^prorrii se 

This is, perhaps, the Rost dangerous threat to objectivity, because 
it nay never appear "on the table." V/hen .he project under consideration is 
the "brain child," "claim to fane," favorite effort, or most popular program 
of the "po.;ers that be," then it nay be understood - perhaps tacitly - that 
the, pronran is an automatic success. The role, of the evaluator is to lend 
scientific dignity to political expediency. 

Dealing with this source of comprorr^ise will prove to be impossible unless 
the Research and Evaluation Office is able to convince the "powers that be*' 
that rnaintaining the professional integrity of the evaluator is in its best 
interests. Here, too, the line relationship can be critical. The most 
advantac;eous position, of course, is v^hen the Research Office has a line to 



the top separate froni that of the program beinc] evaluated. Where this is not 
poss.Y^Me, the top administration r.uist be convinced that the protection of the 
professional integrity of the Rc*=iearch Office is in their best interests. 

Several points can be made regc":rding this proposition. First, it can be 
easily shown that it is a lot cheaper to hire a public relations copywriter 
than to maintain a Research Office. The sinple fact is that if one doss not 
need "truth" he does not need a Research Office. 

Second, rnost adni n i strators do not have technical skills in research de- 
sign or statistics. In a very real v;ay their acceptance of research findings 
is an act of faith. Mov;, if an evaluator will lie for the administrator to 
feather his o\/n nest, it is also possible that he will lie to the administrator 
to feather his own nest. How will the administrator ever know? 

Of course, enlightened administrators might well see evaluation as a 
vehicle for program improvement. If their commitment is to the success of the 
program rather than to the existing form ot the program, then political 
compromise should not arise. 

If top administration is not willing to protect the Research Office from 
political compromise, either for the above reasons or any other, then use of 
an outside consultant may be the only answer. 

3. iQVolvement Compromise 

When a formative evaluation is conducted (though this might not be 
the only case), the evaluator becomes closely involved with project personnel. 
Such a relationship is necessary for his function. The evaluator is continually 
feeding back information on the progress of the project. Part: of the responsi- 
bility of the evaluator is to interpret his findin'^s. Vihere the evaluator has 
a good relationship with the project staff, there is a danger that the Hne 
between making recommendations based on findings and making program decisions 
may wear thin or disappear completely. This leaves the evaluator in the 
position of having to evaluate "hi.^ own baby." 

Dealing with this problem requires an ongoing effort by the evaluation 
supervisor. Through regular supervisory conferences he needs to keep himself 
abreast of the evaluator 's activities and question any that seem to be 
i napproprlate. 

This is not an overly difficult process except in the case of an evaluator 
v'/hose credibility with program personnel is so great that they turn to him for 
solutions to program problems. Here it is necessary for the evaluation super- 
visor to intervene co interpret the role of the evaluator to program personnel. 
The evaluator may explain his data, interpret his data, and discuss its 
Implications for the program being evaluated, but he may not attempt to make 
prograr decisions. 

•Where the evaluator is not a specialist in the area being evaluated (and 
this will certainly be the case with evaluators who must deal with a variety 
o"" projects), it is usually not too difficult to convince both evaluator and 
progran personnel of the pitfalls of role confusion. 



Where the cvaluator is a specialist in the area being evaluated, then 
there is potential for much greater difficulty in avoiding role confusion. 
Consider, for example, the possibilities v/hen a specialist in Early Childhood 
Education v/hose orientation is behavior modification, is evaluating an open 
classroon program! The evaluator r.ay have such strong feelings about what he 
thinks "should be" that it will affect the way he sees what "is." The 
evaluation supervisor must monitor the role of the evaluator to assure that 
he does not compromise his objectivity by beco.Tiing involved in the operation 
of the progranj and decision making about it. 

k. Affective Compromise 

This category is closely related to invob'enent compromise. Often 
the evaluator builds close relationships with project personnel. This can be 
beneficial both in terms of facilitating his information gathering functions 
and in terms of acceptance of his recommendations by project personnel. On the 
other hand, it can easily lead to reluctance on the part of the evaluator to 
Interpret findings in a light that is less than complimentary to the program. 

Though the Research Office evaluates programs and not people, it is people 
and not programs who read the reports. Evaluators who work with program per- 
sonnel over a period of time cannot help having feelings about them - one way or 
the other. Ideally this. relationship has been on a strictly professional level 
and the evaluator is completely objective, but this cannot be guaranteed. It 
is the function of the evaluation supervisor, then, to keep the evaluator in 
perspective. The evaluation supervisor functions as an Internal Auditor to 
assure that the evaluation has been properly conducted, that all of the findings 
have been interpreted, and that all findings are clearly based on data. By 
knov/ing the evaluator, the project manager, and the relationship between the two, 
he Villi know what to expect and will be able to look for it. 

At the same time, the evaluation supervisor can "take the heat" for the 
evaluation findings, should this be necessary. He can sympathize with how hard 
project personnel v;orked and agree that it really isn't fair that test scores 
weren't higher and concede that no test can evej hope to reveal the untold 
benefits of the program, but still insist that the evaluation will be based on 
findings rather than aspirations. He can make it clear that as the cvaluator's 
supervisor, he v/Ill accept nothing less. 

In suTir.ary, the first two sources of compromise - factual compromise and 
political compromise - cannot be accepted. They constitute a compromise of 
integrity. To accept tliem would mean the death (and rightly so) of educational 
cvaulat i on. 

It is suggested that a code of ethics v/hich deals with these issues is 
needed for evaluators. The research and evaluation community should assume as 
a top priority the design and acceptance of such a code of ethics. They should 
enforce It by refusing to recognize any evaluation, evaluator, or Research 
Office which fails to subscribe to it and uphold it. .Moreover, client popula- 
tions {i.e.. Federal and State ''iovernnental Agencies) should be encouraged to' 
approve such a code and make it mandatory for all projects funded, thus 
assuring that the independence of the evaluation is largely a matter of policy. 



The second two sources of cor.iprop^i se , irwolvemcnt compromise and affective 
corupro.Tiise, arise fropi people relating to people. We shall r ver be free of 
thoni. The evaluation supervisor, by being aware of them, can deal with them, 
as suggested above. 

In functioning as an Interna! Auditor, the evaluation supervisor stands 
between the project director and the evaluator. The project director must 
clearly understand the supervisor's line r e 1 at i onsh i p wi th the evaluator. 
(ihere should be no doubts in the evaluator's mind on this issue, either.) 

The evaluation supervisor must be familiar with the project, the project 
director (other project personnel, if possible), and with the evaluator. Only 
in this way will he understand what the relationships involved are and kno\7 
for what to be on the alert. 

All of the above may paint a picture of an aversive relationship. This 
certainly need not be true. As long as all parties under="itand the roles 
i.ivolved and the need for then, there is no reason to expect anything but a 
friendly, professional relationship, especially since program improvement is ~ 
or should be ~ everyone's goal. 
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